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Chapter 1 



INTRODUCTION 



1.1 Background 

One of the major problems facing the construction industry in the coming 
years is its continual decline in productivity. A recurring problem since the mid 
1960s, the extent of the decline continues to elude researchers and industry officials. 
The inability to determine the extent of the decline has been attributed by the Buisness 
Roundtable to the diverse and fragmented nature of the industry, as well as the 
absence of a standard method of measurement and an industry wide data base. 1 In 
examining the productivity issue it is obvious that all aspects of design, construction 
and project management must be reviewed. 

A general consensus exists that as a project progresses from feasibility studies 
and preliminary design, to detailed design, procurement, and construction, the 
amount of control or influence that can be exerted on the project decreases. Decisions 
made early in the design phase have the largest impact on not only total construction 
cost, but often on the constructability, maintainability, and operability of the facility. 
Typically, the cost of the design phase represents less than 10% of the total 
construction costs. Due to their large impact and relatively small cost, it is essential 
that the products of the design phase be examined when productivity improvements 
are desired. 
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The Construction Industry Institute has identified a matrix evaluation 
technique known as the Objectives Matrix as a tool for evaluating the effectiveness of 
design. The matrix format allows the evaluation of many diverse factors that are 
normally difficult to quantify and produces an index which is used to track and 
evaluate performance. 

1.2 Purpose 

The purpose of this study is: 

1. to explore the applications of the design evaluation matrix in 
commercial building and heavy civil construction. 

2. to review guidelines for implementing the design evaluation 
matrix. 

3. to make recommendations for additional criteria and sub-criteria. 



1.3 Scope 

This study is a continuation of a Construction Industry Institute study on the 
evaluation of design effectiveness. Phase I of the research introduced the Design 
Evaluation Matrix, a variation of the Objectives Matrix, as a technique for 
determining design effectiveness. 2 In phase II, the components of the Design 
Evaluation Matrix were refined through a study of the piping phase of large industrial 
projects. 3 The evaluation criteria and sub-criteria used in this study are based on 
those established in phase I of the research. These criteria are: accuracy of the design 
documents, usability of the design documents, cost of the design, constructability, 
performance against design schedule, economy of the design, and ease of occupancy / 



start-up. 
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1.4 Organization 

This report briefly reviews the Design Evaluation Matrix as an evaluation 
technique for design effectiveness. A review of the matrix theory and matrix 
construction is presented in Chapter 2. In Chapter 3 the evaluation criteria and sub- 
criteria are discussed. The feasibility of applying the method to commercial building 
and heavy civil construction is explored in Chapter 4. A Design Evaluation Matrix 
example for a commercial building is presented in Chapter 5 using the weights and 
performance levels collected from the various projects volunteered for the study. 
Chapter 6 presents the conclusions of the study and recommendations for the report. 
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Chapter 2 



THE DESIGN EVALUATION MATRIX 

In an attempt to improve productivity throughout the construction industry 
many Industrial Engineering techniques used in manufacturing industries such as time 
and motion studies and time-lapse photography, have been adapted for use by the 
construction industry. Another method receiving increasing attention is matrix 
measurement. Matrix measurement does not direcdy measure productivity. Instead, 
the factors that affect production time, quality, and cost are measured. One form of 
matrix measurement known as the Objectives Matrix provides the means to list, 
categorize, and weight the key indicators of performance that relate to the objectives 
of a project. A variation of this method, called the Design Evaluation Matrix, is being 
researched by the Construction Industry Institute for evaluating design effectiveness. 

2.1 Introduction 

There are four main components of an evaluation matrix, the criteria (sub- 
criteria), the weights, the performance scale, and the performance index. 

The criteria (sub-criteria) define what is to be measured. The weights 
determine the relative importance of the criteria (sub-criteria) to the operation of 
interest. The performance scale compares the measured value of a specific criterion 
on a project to past performance or to future goal for that criterion. From these three 
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components the performance index is calculated and used to evaluate and track 
performance. 

2.2 Construction of The Matrix 

This section is provided a guide on how to construct an evaluation matrix, and 
to illustrate how the components of the matrix interact. Figure 2. 1 should be referred 
to throughout the explanation of matrix construction. 

The first step in constructing a matrix is the selection of the evaluation criteria. 
In developing the Objectives Matrix, Riggs' proposed three guidelines for selecting 
evaluation criteria. 1 First, the criteria should pertain only to the activities that 
contribute directly to the performance goals. Second, all goals should be reasonable, 
and their attainment dependent only upon the actions of the group being measured. 
Finally, the criteria should be representative of all of the work responsibilities of the 
group. 

Once the criteria (sub-criteria) have been selected weights are assigned by 
management in accordance with its perception of how each criterion impacts 
performance. In the weighting process, 100 points are distributed among the criteria. 

Notice that a performance scale of 0 to 10 has been drawn on the right side of 
the matrix. A score of 10 represents a future goal that is attainable in the foreseeable 
future with current resources. A score of 3 is designated as average (rather than a 
score of 5) to allow more room for improvement. A score of 0 represents the 
minimum acceptable level of performance based on recent experience. These three 
levels represent the "benchmark" levels for the matrix. 

After the values for the benchmark scores are established, values are 
determined for the levels between benchmarks. These values may be established by 
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one of three different techniques, numerical increment, percentage increment, or a 
subjective ratio. The technique used will depend on the nature of the criterion and the 
workers being evaluated. 

In the numerical increment technique each level on the performance scale 
differs by the same numerical value. For example, the increment between scores one 
and two is equal to the increment between scores eight and nine. These numerically 
linear increments result in an increasingly larger percentage of improvement with each 
upward movement. A formula to determine the numerical increment for values 
between the benchmark levels is presented in Appendix 1. 

The percentage increment technique utilizes a non-linear scale. This technique 
produces increments in which the percentage difference between any two consecutive 
levels is the same. A formula to determine the percentage increment scale is also 
contained in Appendix 1 . 

After all values have been entered in the appropriate columns and rows of the 
matrix, the project scores assigned to each criterion are entered in the row under the 
criteria labeled "performance". 

In each column the score closest to the project score that does not exceed it is 
circled. If a measured score falls between matrix scores it is assigned the lower of the 
two scores. A score is not assigned until it has been attained. 

The circled scores are recorded in the box below each criterion labeled 
"score". This score is multiplied by the criterion weight, and the product is recorded 
in the row labeled "value". After all values have been determined they are added 
together to obtain the "performance index". 
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2.3 Sub-Matricies 

The use of matrix measurement allows each criterion on the master matrix to 
be individually measured by the use of sub-matricies. A sub-matrix is developed for 
each criterion on the master matrix in the same manner that the master matrix was 
developed. Sub-criteria are much more specific than the performance criteria on the 
master matrix. Once completed, the performance index of a sub-matrix becomes the 
measured value for the appropriate criterion on the master matrix. This enables the 
user to track the performance of a specific criterion by tracking the performance index 
of the corresponding sub- matrix. 

An example of a sub-matrix and its relationship to a criterion on the design 
evaluation matrix is shown in Figure 2.2. 
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Chapter 3 



DESIGN EVALUATION CRITERIA 

In order for the design evaluation matrix to be utilized effectively, evaluation 
criteria must be identified that relates to the design function and to potential 
construction and maintenance problems that may result from an improper design. In 
its effort to develop the design evaluation matrix, the Construction Industry Institute 
has identified fourteen evaluation criteria that are generally applicable to all types of 
construction projects. These evaluation criteria were provided to owners, 
constructors and designers as a starting point for this investigation. 

3.1 Criteria Selection 

Design is a complex function and will require a variety of performance criteria 
to effectively evaluate it. Many variables will influence the actual selection of design 
criteria and sub-criteria. Since each project is unique to a degree, the selection of 
criteria needs to be tailored to each project. Variables influencing criteria selection 
include the type of project, type of design contract, type of construction contract, 
scheduled duration of the design and construction periods, and the project's budget. 
Additionally, the differing perspectives of the designer, constructor, and owner need 
to be considered in selecting evaluation criteria and sub-criteria. Some of the 
designer's principal concerns are a design that meets all owner requirements and 
regulatory codes, is aesthetically pleasing, and enhances its professional reputation. 



12 



13 



The constructor is concerned with the constructability of the design, the accuracy of 
the design products, and its ability to meet the owner's schedule, as well as the ability 
to generate a profit. The owner's concerns include the items mentioned above along 
with the long range aspects of the design, such as life-cycle costs, maintainability, 
and operability. 

3.2 Design Evaluation Criteria 

During the initial phase of its study, the Construction Industry Institute 
identified fourteen design criteria. 1 With only minor modifications these criteria can 
be utilized on most construction projects. These criteria are presented in Figure 3.1. 



Design Evaluation Criteria 



Accuracy of the Design Documents 
Completeness of the Design Documents 
Clarity of the Design Documents 
Usability of the Design Documents 
Economy of the Design 
Timeliness of the Design 
Start-up Costs and Time Required 
Cost of the Design Effort 
Constructability of the Design 
Operability of the Design Facility 
Maintainability of the Design Facility 
Safety of the Design 
Plant Operating Efficiency 
Plant Performance 



Figure 3. 1 



The process of design evaluation should be an ongoing endeavor and can be 
divided into three phases. In phase one, efforts are made to track the effectiveness of 
the design during the design process. The injection of contractor experience and 
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knowledge can play a significant role in improving the constructability and cost 
effectiveness of a design. 

In the second phase, construction, the design is evaluated for constructability, 
accuracy, clarity, and information content. Studies have shown a direct relationship 
between clarity, information content, and overall quality of a design, and the 
productivity of construction workers . 2 

Immediately after construction is completed an effort should be made to gather 
information from all members of the project team prior to its dispersion. The 
cataloging of this information can provide a valuable library of "lessons learned" and 
prevent the repetition of errors on future projects. 

In phase three, a long term evaluation needs to be performed to evaluate the 
overall effectiveness of the design. Several aspects which should be included in this 
evaluation are the maintainability of the facility, layout efficiency, and quality. 

Based on discussions with industry personnel, those criteria which lend 
themselves to an initial design evaluation are: Accuracy of the Design Documents, 
Usability of the Design Documents, Cost of the Design, Constructability of the 
Design, Economy of the Design, Performance Against Schedule, and Ease of 
Occupancy. These criteria are discussed in the following paragraphs. 

The most obvious products of the design process are the drawings and 
specifications. The accuracy of the design documents criterion evaluates the 
effectiveness of the design by monitoring the frequency and impact of errors and 
omissions in the drawings and specifications. Examples of quantitative 
measurements for this criterion include the number of drawing revisions, the number 
of revised drawings, the number of specification revisions, the number of revised 
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specifications, and the number of rework manhours due to design errors and 
omissions. 

The usability of the design documents measures the completeness, clarity and 
information content of the plans and specifications. Subjective measurements for this 
area include the amount of cross referencing, the appropriateness of the drawing size, 
the completeness of the drawings, and the clarity of the drawings. 

The cost of the design effort evaluates the anticipated design cost versus the 
actual design costs. Caution must be used in monitoring this criterion as many factors 
may impact on the number of design hours and cost of the design . Owner changes, 
regulatory (code) changes and schedule compression are some examples of factors 
that may impact on the design cost and duration. These factors should be accounted 
for through a revision of the design budget and schedule. 

Constructability is defined as the optimum integration of construction 
knowledge and experience in planning, engineering, procurement and field operations 
to achieve overall project objectives . 3 General areas indicating the effectiveness of a 
project's constructability program include the type of construction equipment 
required, size and skill of the labor force required, special material requirements, and 
unrealistic tolerances. 

The economy of design criterion attempts to evaluate the efficiency of the final 
design product. Indicators of an efficient design include minimizing the number of 
overdesigned structural members and overspecified materials, and the efficiency of 
the building layout. 

One of the critical factors in the success of a construction project is the 
availability of the design. Delays in the release of the design documents can 
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adversely affect not only the project's overall design and construction schedule, but 
may increase the cost of bids by contractors, vendors and material suppliers. 

The ease of occupancy and start-up criterion addresses the completeness of the 
design. The efficiency of the design process is judged by comparing the budgeted 
manhours to the actual manhours required to prepare the facility for occupancy or use. 

These seven criteria should be of concern to each of the design users. 
Additionally, with only limited modifications they can be used for almost any type of 
construction project. Before their final selection for use in a matrix, each criterion 
should be thoroughly reviewed not only for its applicability to type of project, but 
also to determine how it will be measured, subjectively or quantitatively. 

3.3 Criteria Weighting 

Each of these criteria will vary in the degree of their accuracy as an indicator 
of design effectiveness. To compensate for this, each criterion is weighted by the 
design user to reflect the impact of the criterion in relation to the other criteria. The 
number of points assigned to a criterion is a direct indication of its influence on a 
design’s effectiveness. 

In weighting of the criteria consideration must also be given to the uniqueness 
of the individual project. A project that is schedule driven may require special 
emphasis on the performance against schedule criterion and constructability. A 
project that is on an extremely tight budget may place additional emphasis on the cost 
of the design and economy of the design. 

Finally, the objectiveness or subjectiveness of a criterion may influence its 
weight. Due to a possible lack of confidence in, or controversy surrounding 
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subjective ratings, a user of the matrix may wish to emphasize those criteria that are 
measured quantitatively. 
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Chapter 4 



APPLICATION OF THE DESIGN EVALUATION MATRIX TO 
COMMERCIAL BUILDING AND HEAVY CIVIL PROJECTS 

To explore the application of the Design Evaluation Matrix on commercial 
buildings and heavy civil construction projects, it was necessary to obtain data from 
actual projects and solicit the views of construction industry professionals. 

Since the application of the Evaluation Matrix is relatively new, the seven 
initial criteria for measurement of design effectiveness were used. Industry personnel 
were encouraged to suggest additional criteria and sub-criteria for use in the 
evaluation matrix. 

4.1 Methodology 

The first step in testing the feasibility of the evaluation matrix was the 
collection of project data. Questionnaires were mailed out to CII member companies / 
firms requesting projects for the study. 

An interview guide was included with each project request letter to aquaint 
personnel designated as "points of contact" with the purpose of the study prior to 
interviews. The design evaluation matrix was presented with a description of the 
criteria and sub-criteria. Efforts were made to contact owners, designers, and 
constructors of commercial buildings and heavy civil construction projects. 
However, no responses received from design firms. 
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Telephone interviews were conducted with personnel from those companies 
responding to the request for projects. The projects volunteered for the study ranged 
from a 430,000 SF office complex to a power plant facility including 1500 feet of 
tunnels connecting the major buildings. All project personnel interviewed were 
degreed engineers. 

The interview guide was divided into four sections. The first section provided 
a summary of the research project, its scope, and requested a point of contact. An 
explanation of the design evaluation matrix was presented, criteria and sub-criteria 
discussed, and definitions provided. 

Section two contained a project description sheet requesting general 
information about the project. Also included in this section was an interview guide 
addressing each of the seven design criteria, and requesting the information required 
for determining the sub-criteria ratios. 

In section three the interviewees were asked to rank the criteria and sub- 
criteria in order of their importance, and to assign weights to the individual criteria 
and sub-criteria. 

In section four the interviewees were asked to estimate the industry averages 
for the benchmark performance levels of ten, three, and zero for each of the sub- 
criteria. These are subjective estimates based on the cumulative experience and 
knowledge of the interviewees. 

4.2 Sources of Data 

Letters requesting projects for the study were sent to representatives of 
owners, designers, and constructors belonging to the Construction Industry Institute. 
Projects were requested in the categories of commercial buildings (office buildings, 
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corporate headquarters etc.) and heavy civil projects (highway, bridge, and the heavy 
civil phase of industrial projects, etc.). 

Responses were received from five firms, representing owners (3) and 
constructors (2). No response was received from a design firm. One constructor, 
who was also the facility's owner, is classified as an owner. The projects reviewed 
included three commercial buildings and two heavy civil projects. Additionally, one 
constructor provided assistance in the determination of benchmark values, industry 
weights, and recommendations for additional criteria and sub-criteria in the 
commercial building sector. Figure 4. 1 provides a brief description of each project 
volunteered. 



Project Summary 



Commercial Buildings 



Classification 

Owner 



Project Description 

Regional headquarters (300,000SF), offices, 
conference facilities, and cafeteria. 



Owner 



Regional headquarters (430,000 SF), 
conference facilities, computer center, and 
cafeteria. 



Owner (Constructor) 



Corporate office and laboratory expansion. 



Constructor 



* * * * 



Figure 4. 1 
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Project Summary (Continued) 



Heavy Civil 




Classification 


Protect Description 


Constructor 


Office building, refinery power plant, facilities 
building, 1500 feet of interconnecting tunnels. 


Constructor 


Nuclear waste processing plant (new 
technology) 



Figure 4. 1 



4.3 Design Criteria 

The initial step in evaluating design effectiveness is the identification of sub- 
criteria for each of the design criteria. The sub-criteria should be much more specific 
that the criteria on the master matrix. 

In an effort to increase the acceptance of the evaluation matrix sub-criterion 
should be quantitatively measured. Subjective measurements should be used only 
when absolutely necessary. 

The sub-criterion for each of the seven criteria are shown in Figure 4.2: 
Criteria and Sub-Criteria. The abbreviated criteria and sub-criteria titles utilized in the 
matricies are contained in parentheses. 
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Criteria and Sub-Criteria 



ACCURACY OF THE DESIGN DOCUMENTS (ACCURACY) 

- Drawing revisions / total number of drawings (Drawing Revisions) 

- Revised drawings / total number of drawings (Revised Drawings) 

- Specification revisions / total number of specifications (Specification Revisions) 

- Revised Specifications / total number of specifications (Revised Specifications) 

- Manhours of rework / total number of manhours (Rework) 

USABILITY OF THE DESIGN DOCUMENTS (USABILITY) 

- Subjective rating for drawing size (Drawing Size) 

- Subjective rating for number of drawings (Number of Drawings) 

- Subjective rating for number of cross references (Cross References) 

- Field engineering manhours / total engineering manhours (Field Engineering) 

- Subjective rating for clarity of drawings (Clarity) 

- Subjective Rating for completeness of drawings (Completeness) 

COST OF DESIGN EFFORT (COST) 

- Design cost / construction cost (Design Cost) 

- Design manhours expended / design manhours budgeted (Design Manhours) 
CONSTRUCTABILITY (CONSTRUCTABILITY) 

- Subjective rating for the number of unrealistic tolerances vs. quality expectations 
(Tolerances) 

- Subjective rating for the number of crafts (Different Crafts) 

- Subjective rating for compatibility with current materials and technology 
(Compatibility) 

ECONOMY OF DESIGN (ECONOMY) 

- Building layout efficiency (Layout) 

- Amount of overdesigned members (Overdesign) 

- Amount of overspecified materials (Overspecified) 

PERFORMANCE AGAINST SCHEDULE (SCHEDULE) 

- Percent of construction document release dates attained (Document Release) 

- Percent of intermediate release dates attained (Intermediate Release) 

EASE OF OCCUPANCY (OCCUPANCY) 

- Occupancy preparation days required / days budgeted (Occupancy Days) 

- Number of maintenance personnel required / number budgeted (Personnel) 



Figure 4.2 
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The accuracy of the design documents criterion concerns the number of errors 
and omissions in the plans and specifications. For commercial buildings and heavy 
civil projects, as with most construction projects, accuracy is best reflected by the 
number of revisions in the design documents, and the number of rework manhours 
due to these errors and omissions. In evaluating the accuracy of the design 
documents only those drawings issued for construction are reviewed. Prior to 
issuance, a drawing is not considered to be complete. Sub-criteria utilized in 
evaluating the accuracy criteria include distinctions between the number of drawing 
revisions and the number of revised drawings, and the number of specification 
revisions and the number of revised specifications. This was necessary since the 
impact of numerous revisions to a single drawing or specification may result in a 
different impact when compared to a larger number of drawings or specifications 
being revised. 

Usability of the design documents is primarily a subjective criterion that 
reflects the information content and format of the drawings and specifications. The 
use of subjective ratings in this and other criteria must be approached with caution due 
to the different education, experiences, and standards of each evaluator. The only 
quantitative sub-criteria included in this evaluation is the number of field engineering 
manhours versus the total number of design manhours. 

The cost of the design can be subdivided for the various phases (electrical, 
mechanical, HVAC, etc.) of a project to determine the overall cost effectiveness of 
the project. The actual cost versus the budgeted costs is one measure of the 
effectiveness of the design effort. Due to the preliminary nature of this study no 
subdivisions were made. Allowances should also be made for owner initiated 
changes affecting scope, installed equipment, materials, and methods. 
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The constructability criterion is subjectively rated and difficult to measure due 
to the unique aspects of each project’s program. Some factors that impact on 
constructability include unrealistic tolerances which can increase costs and slow 
production, and the type of equipment to be installed which when coupled with the 
number of different crafts required may increase the probability of jurisdictional 
disputes on union projects. Other important items to be reviewed include the use of 
prefabrication yards, models, taking advantage of repetitive operations to obtain the 
maximum benefits from learning curves, and the necessity of specialized equipment 
and skills. 

The economy of design criterion is indicative of a design's waste and 
inefficiency. Material waste adds directly to the cost of a project and operational 
inefficiencies may create scheduling and access problems. Additionally, the presence 
of overdesigned members and overspecified materials can increase the cost of a 
project. 

For a project to remain on schedule the design documents must be issued on 
their scheduled release dates. Poor performance in the release of plans and 
specifications can impact on areas such as procurement, quality, and schedule. 

The ease of occupancy and start-up criterion attempts to evaluate the 
completeness of the design and identify unexpected problem areas. An increase in the 
maintenance manhours required prior to occupancy or use may be indicative of a 
design omission. 

During the interviews industry personnel were asked to suggest additional 
criteria and sub-criteria to add to the matrix. Suggested additions to the seven criteria 
for the master matrix are: 



1 . Aesthetics 
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2. Maintainability 

3. Installed systems reliability. 

4. Expandability 

5. Quality 

6. Durability of finishes 

7. Safety 

Recommendations for additional sub-criteria included: 

Accuracy of the Design Documents 

1 . The number of drawing clarification memos sent to the designer by the 
contractor / the total number drawings. 

2. Number of changes due to misunderstanding of the building code 
requirements. 

3. Number of changes due to errors and omissions and their associated 
costs. 

Usability of the Design Documents 

1. The number of requests for information sent to the designer by the 
contractor. 

2. Number of changes due to shop drawing review comments. 

Cost of the Design Effort 

1. Duration of the design. 

Constructability 



1. Engagement of the contractor. 



27 



Maintainability 

1. Accessibility of building utilities / services (e.g. plumbing, HVAC, 
piping, ducts, controls, etc.) for maintenance and repair. 

2. Maintenance manhours budgeted vs. maintenance manhours expended. 

3. Actual cost of maintenance vs. budgeted costs. 

4. Quality of documentation (as-built drawings). 

5. Durability of materials chosen for high traffic areas. 

4.4 Weights 

The evaluation criteria used in the Design Evaluation Matrix were assigned 
weights to reflect their impact on the effectiveness of design. These weights, based 
on the averages of the weights assigned by industry personnel, are shown in Figure 
4.3: Criteria Weights. 



Criteria Weights 



Criteria 


Commercial Buildine 


Heaw Civil 


Accuracy 


20 


20 


Usability 


20 


20 


Cost 


10 


10 


Constructability 


20 


25 


Economy 


10 


10 


Schedule 


15 


10 


Occupancy 


5 


5 



Figure 4.3 
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For commercial buildings and heavy civil projects, the criteria accuracy of the 
design documents, usability of the design documents, and constructability, were 
ranked as the most important indicators of design effectiveness. A slightly higher 
weight was assigned to constructability by the heavy civil evaluators than by those 
evaluating commercial buildings. While assigned equal weights, the top three 
evaluation criteria for commercial buildings in rank order, were, constructability, 
accuracy of the design documents, and usability of the design documents. 

The sub-criteria for each criterion were also assigned weights to reflect their 
relationship to the evaluation criterion on the master matrix. These assignments are 
presented in Figures 4.4 through 4.10. 

On both commercial building and heavy civil projects, the amount of rework 
was considered to be the most important indicator of design document accuracy. This 
was expected due to the relationships between rework, cost, productivity, schedule, 
and worker morale. The number of revised specifications was rated the least 
important sub-criteria for heavy civil projects. 



Accuracy Sub-Criteria Weights 



Criteria 

Drawing Revisions 
Revised Drawings 
Specification Revisions 
Revised Specifications 
Rework 



Commercial Building 



15 

20 

15 

15 

35 



Heavy Civil 



15 

15 

20 

10 

40 



Figure 4.4 
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The usability of the design documents was ranked as the third most important 
evaluation criteria on both commercial building and heavy civil projects. The sub- 
criteria, completeness, and clarity, were judged the most important indicators of 
design document usability for both types of construction. 

Usability Sub-Criteria Weights 



Criteria 


Commercial Buildins 


Heaw C 


Drawing Size 


5 


5 


Number of Drawings 


10 


5 


Cross References 


15 


10 


Field Engineering 


15 


10 


Clarity 


25 


35 


Completeness 


30 


35 



Figure 4.5 

The cost of the design criterion was generally ranked low in importance. Of 
the sub-criteria, personnel involved with heavy civil projects favored design 
manhours over design cost. One interesting response was the assignment of an 
extremely high weight to a write-in criteria, duration of the design process, by one of 
the evaluators for commercial buildings. 
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Cost Sub-Criteria Weights 



Criteria Commercial Building Heavy Civil 

Design Cost 75 40 

Design Manhours 25 60 



Figure 4.6 

Overall, the constructability criterion was rated as the most important indicator 
of design effectiveness. In evaluating the constructability sub-criteria, both industry 
segments rated the amount of unrealistic tolerances as the most important factor. 
Compatibility of the design with current materials and technology was ranked second 
by both groups. 



Constructability Sub-Criteria Weights 



Criteria 


Commercial Buildine 


Heavy Civil 


Tolerances 


50 


60 


Different Crafts 


10 


10 


Compatibility 


40 


30 



Figure 4.7 
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Economy Sub-Criteria Weights 



Criteria 

Layout 

Overdesign 

Overspecified 



Commercial Buildine 


Heaw Civil 


55 


15 


25 


45 


20 


40 



Figure 4.8 

For commercial building projects the sub-criteria layout was favored by a 
wide margin. It is interesting to note that both overdesigned members and 
overspecified materials were weighted very close together for heavy civil projects, 
while layout received relatively little attention. 

Schedule Sub-Criteria Weights 



Criteria Commercial Building Heavy Civil 

Document Release 55 45 

Intermediate Release 45 55 



Figure 4.9 

For the performance against schedule criterion, both sub-criteria were 
regarded as relatively equal. Those personnel evaluating commercial buildings 
favored document release dates met, while the heavy civil evaluators favored 
intermediate document release dates. 
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Occupancy Sub-Criteria Weights 



Criteria 

Occupancy 

Personnel 



Commercial Building 



Heavy Civil 



55 

45 



55 

45 



Figure 4.10 



The occupancy / start-up criterion was ranked the least important on both 
commercial and heavy civil projects. Weight assignments for the sub-criteria were 
identical with the number of days required to prepare for occupancy / put the facility 
into use favored over the number of manhours required to prepare the building / 
facility. 

4.5 Performance Scale Assignments 

The assignment of benchmark values by industry personnel are shown in 
Figures 4.11 and 4.12, Estimated Benchmark Values. These are the values that 
would be entered into the master and sub-matricies at the performance levels of ten, 
three, and zero. The increments between these scores would be determined by 
utilizing the percentage increment technique, numerical increment technique, or the 
subjective ratio technique. Caution must be used in applying these values to actual 
projects due to the limited number of projects studied during this investigation. 

Figure 4.11 presents the estimated benchmark values for commercial 
buildings based on the projects volunteered for the study. The "Measured Value" 
represents the average of the values assigned to a particular sub-criteria by industry 
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Estimated Benchmark Values 
Commercial Buildings 



Percentage Values Measured Value 

Sub-Criteria Ten Three Zero 



ACCURACY 



Drawings Revisions 


5 


25 


50 


35 


Revised Drawings 


5 


25 


50 


46 


Specification Revisions 


5 


25 


50 


40 


Revised Specifications 


5 


25 


50 


9 


Rework 


2 


5 


15 


2.3 


USABILITY 


Field Engineering 


5 


30 


55 


35 


COST 


Design Cost 


4 


7 


9 


9 


Design Manhours 


100 


120 


140 


116 


SCHEDULE 


Document Release 


100 


80 


65 


68 


Intermediate Release 


100 


80 


70 


64 


OCCUPANCY/ START-UP 


Occupancy 


90 


110 


130 


92 


Personnel 


90 


115 


130 


127 



Figure 4.11 

personnel for the projects volunteered for the study. All values are expressed as 
percentages. A comparison of the measured values with the estimated values 
indicates that the average project is generally between the level three and level zero 
benchmarks. This is even more apparent when the measured values are placed in the 
example evaluation matrix in the next chapter. 
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Figure 4.12 presents the estimated benchmark values for heavy civil projects 
based on the projects studied. A comparison of project measured values with the 
estimated benchmark values indicates that the average heavy civil project falls between 
the level three and level zero benchmarks. The measured values for two of the sub- 

Estimated Benchmark Values 
Heavy Civil 



Sub-Criteria 


Percentage Values 
Ten Three Zero 


ACCURACY 


Drawings Revisions 


20 


225 


400 


Revised Drawings 


10 


50 


85 


Specification Revisions 


10 


50 


85 


Revised Specifications 


2 


10 


25 


Rework 


2 


10 


25 


USABILITY 


Field Engineering 


3 


12 


25 


COST 


Design Cost 


10 


20 


30 


Design Manhours 


100 


120 


130 


SCHEDULE 


Document Release 


100 


75 


60 


Intermediate Release 


100 


70 


60 


OCCUPANCY/ START-UP 


Occupancy 


100 


110 


120 


Personnel 


100 


110 


120 



Measured Values 



350 

75 

340 

60 

6 



23 



24 

125 



50 

55 



Figure 4.12 



100 

100 
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criteria, specification revisions and revised specifications score substantially above 
the zero level of performance. These values are heavily influenced by the new, 
unproven technology associated with one of the heavy civil projects studied. 
Sufficient data is not currently available to determine the actual values that should be 
used for the benchmark scores. These values are provided for illustrative purposes 
only. 



4.6 Conclusions 

The Design Evaluation Matrix is extremely flexible and holds a great deal of 
promise for use throughout the construction industry. The information required to 
construct the matricies is readily available on current construction projects. Due to the 
limited number of projects investigated during this study, the values presented in this 
paper should be considered for illustrative purposes only. As the evaluation matrix 
receives more attention from industry the database for criteria and sub-criteria weights 
and benchmark levels should expand and become more refined. 



Chapter 5 



DESIGN EVALUATION MATRIX EXAMPLE 

The next step in exploring the application of the Design Evaluation Matrix to 
the areas of commercial buildings and heavy civil construction is the construction of 
an example matrix. In this chapter, an evaluation matrix for a commercial building 
will be constructed and discussed. 

5.1 Construction of the Matricies 

The first step in the construction of the master matrix and sub-matricies is 
entering the criteria and sub-criteria column headings, the weight assignments, and 
the level zero, three, and ten benchmark estimates in the matrix forms. For each 
criterion in the master matrix and sub-criterion in the sub-matricies, the benchmark 
values collected during the interviews will be used. 

After all the interview data has been entered in the matricies scores must be 
assigned to the increments between the benchmark levels of zero, three, and ten. In 
this example, the percentage increment technique will be utilized to determine the 
scores for the sub-criteria rework, design cost, design manhours, document release, 
intermediate release, and occupancy. The sub-criteria scores for drawing revisions, 
revised drawings, specification revisions, field engineering, and maintenance 
manhours will be determined using the numerical increment method. Subjectively 
rated sub-criteria scales are equal to their scores. 
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In order to determine the performance index for each of the evaluation criteria 
on the master matrix the sub-matricies must first be completed. The performance 
scores, representing the averages of the data collected during the study, are entered in 
the row labeled performance immediately below the sub-criteria headings. The score 
closest to these values in the sub-matrix body are circled and recorded in the row 
marked "score" at the bottom of the matrix form. Scores are multiplied by their 
corresponding weights, and the results entered in the row labeled "value". The values 
for each sub-criteria are added to yield the "performance index". 

The sub-criteria performance indicies are transferred to the master matrix form 
and recorded under the appropriate criteria heading in the row marked "performance". 

The score that is equal to or less than the performance score is circled in the 
body of the matrix and entered at the bottom of the matrix in the row labeled "score". 
These scores are multiplied by their corresponding weights and the results are 
recorded in the "value" row. The values are added together to produce the overall 
"performance index". 

5.2 Matricies Analysis 

The completed sub-criteria evaluation matricies and the master matrix for the 
example are presented in Figures 5.1 through 5.8. Each will be discussed in the 
following paragraphs. 

The sub-matrix for Accuracy of the Design Documents is shown in Figure 
5.1. Accuracy of the Design Documents was ranked as the second most important 
criterion. Based on the data utilized for this example, the sub-matrix has a 
performance index of 465. While this index is above average, the majority of its 
value comes from two sub-criteria, rework, and revised specifications. Thus, 
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attention should be directed to the sub-criteria revised drawings, drawing revisions, 
and specification revisions. Each of these sub-criteria have extremely low scores, 
with the sub-criteria, revised drawings, contributing zero points to the index. 

Figure 5.2 is the Usability of the Design Documents Sub-Matrix. With a 
performance index of 240, this sub-matrix is considered below average (300). Two 
of the six sub-criteria were rated average, and one, number of drawings, was rated 
above average. However, the most important sub-criterion, completeness of the 
design documents, received the lowest actual score. This tends to reemphasize the 
problems identified in the Accuracy of the Design Documents Sub-Matrix concerning 
the number of drawings and specification revisions, and the number of revised 
drawings. Another interesting observation in this sub-matrix concerns the scores 
associated with those sub-criteria which were rated subjectively. In previous 
research, subjectively rated criteria consistently received scores higher than 
quantitatively rated sub-criteria. This trend was not encountered during this 
investigation. 

The Cost of the Design Effort Sub-Matrix is shown in Figure 5.3. The cost 
of the design effort was given the second lowest priority by industry personnel. With 
a combined score of 100, it has the second lowest performance index. 

Constructability was ranked as the most important design evaluation criterion. 
The Constructability Sub-Matrix is presented in Figure 5.4. As the most important 
criterion, the sub-matrix's performance index of 180, well below average, should be 
of major concern. The most important sub-criteria, unrealistic tolerances versus 
quality expectations, has tied for the lowest score. Though this study was limited in 
the number of projects investigated, this particular sub-matrix illustrates the 
importance and potential of constructability programs. 
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Figure 5.5 is the Economy of Design Sub-Matrix. One interesting aspect of 
this sub-matrix is the low score of the layout sub-criteria. As mentioned earlier, no 
responses were received from design firms. The performance index of 345 is slightly 
above average. 

Figure 5.6 presents the Performance Against Schedule Sub-Matrix. All 
contributors to the study were unhappy with the schedule performance of the design 
documents. Comments form one constructor indicated that each time a scheduled 
release or intermediate release date was missed, it was simply rescheduled as if 
nothing had happened. The performance index of 0 is the lowest of any sub-matrix. 

The Ease of Occupancy / Start-up Sub-Matrix is presented in Figure 5.7. This 
criterion was assigned the lowest weight by industry evaluators. The performance 
index for this sub-matrix was 495, slightly above average. Owners generally were 
able to occupy their projects on or slightly ahead of schedule, but required extra 
personnel to prepare the facilities. 

Figure 5.8, The Design Evaluation Matrix, presents a completed master 
matrix based on the values (performance indicies) obtained from the sub-matricies. 
The overall performance index for this example is 200, below average. While a 
below average index is disappointing, the example illustrates the usefulness of the 
method in identifying problem areas. Attention should be directed to those criteria 
which scored below in their respective sub-matricies, usability of the design 
documents, cost of the design effort, constructability, and performance against 
schedule. Additionally, due to the influence of the accuracy of the design documents 
on these criteria, efforts should be made to improve the performance of this criterion. 
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5.3 Conclusions 

Based on this example matrix evaluation, several conclusions can be reached. 
The data required to evaluate design effectiveness on both commercial building and 
heavy civil construction projects is readily available. While most people interviewed 
felt more comfortable with quantitative measurements, no identifiable score inflation 
occurred due to the use of subjective ratings. The application of the matrix to an 
entire project is rather simplistic. A more extensive study will be required to 
determine the actual weights and benchmark values that should be assigned to the 
various criteria and sub-criteria. Each project should be sub-divided into the major 
design divisions for a detailed analysis. All users of the design, owners, designers, 
and constructors need to be included in the collection of data for a realistic database. 



Chapter 6 



CONCLUSIONS AND RECOMMENDATIONS 

This report investigates the feasibility of utilizing a matrix evaluation technique 
known as the Design Evaluation Matrix, as a method to evaluate the design 
effectiveness in commercial building and heavy civil construction. The Design 
Evaluation Matrix, a variation of the Objectives Matrix, has proven to be a flexible 
and effective method to categorize and weight those factors that have a significant 
impact on a construction project. This final chapter presents the conclusions of the 
study and recommendations for future research. 

6.1 Conclusions 

1 . The seven design evaluation criteria identified in the initial phase of the 
research are applicable to all construction projects regardless of industry 
segment. 

2. The Design Evaluation Matrix can be used during the individual design, 
construction, and operation phases of a project, or combined for a more 
simplistic evaluation of an entire project. 
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3. The list of evaluation criteria should be expanded. This will occur as a 
by-product of the method's utilization by industry, and feedback being 
provided to researchers. 

4. The data collected for this study was inconclusive (due to the limited data 
available) for determining the values that should be used by industry for 
weighting the criteria (sub-criteria) and establishing benchmark values. 

6,2 Recommendations 

1. An extensive database should be established for each segment of the 
construction industry, reflecting the views of owners, designers, and 
constructors. 

2. Data collections should be undertaken to establish benchmark values for 
each major design function (electrical, mechanical, HVAC, et c.). 

3. As the method is adopted for use by industry, feedback should be 
collected to further refine the database. 

4. The experiences of companies utilizing the method need to be cataloged in 
a "lessons learned" format in order to periodically review the guidelines 
for applying the method. 
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5. The additional design evaluation criteria and sub-criteria summarized in 
Figure 6.1 should be considered for future evaluations of commercial 
building and heavy civil projects. 
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Additional Criteria and Sub-Criteria 



Evaluation Criteria 

1 . Aesthetics 

2. Maintainability 

3. Installed systems reliability. 

4. Expandability 

5. Quality 

6. Durability of finishes 

7. Safety 

Sub- Criteria 

Accuracy of the Design Documents 

1 . The number of drawing clarification memos sent to the designer by the 
contractor the total number drawings. 

2. Number of changes due to misunderstanding of the building code 
requirements. 

3. Number of changes due to errors and omissions and their associated 
costs. 

Usability of the Design Documents 

1 . The number of requests for information sent to the designer by the 
contractor. 

2. Number of changes due to shop drawing review comments. 

Cost of the Design Effort 

1. Duration of the design. 

Constructability 

1 . Engagement of the contractor. 

Maintainability 

1 . Accessibility of building utilities / services (e.g. plumbing, HVAC, 
piping, ducts, controls, etc.) for maintenance and repair. 

2. Maintenance manhours budgeted vs. maintenance manhours expended. 

3. Actual cost of maintenance vs. budgeted costs. 

4. Quality of documentation (as-built drawings). 

5. Durability of materials chosen for high traffic areas. 



Figure 6. 1 



APPENDIX 1 



INCREMENT FORMULA TECHNIQUE 
PERCENT FORMULA TECHNIQUE 
SUBJECTIVE RATIO TECHNIQUE 
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AU Numerical Increment Technique 

The following formula can be utilized to determine the numerical increment 
between benchmark levels: 

Numerical Increment Formula 
(X- Y)/Z = I 

X - represents the larger benchmark score ratio 
Y - represents the smaller benchmark score ratio 

Z - represents the number of increments between the benchmark scores, either 
7 (between levels three and ten) or 3 (between levels zero and three). 

I - represents the numerical ratio increment between two consecutive levels 
going 

from Y to X. 
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A1.2 Percentage Increment Technique 

The following formula can be utilized to determine the percentage increment 
between benchmark levels: 

Percentage Increment Formula 
( Y/X)Vz = k 

X - represents the larger benchmark score ratio 
Y - represents the smaller benchmark score ratio 

Z - represents the number of increments between the benchmark scores, either 
7 (between benchmark levels three and ten) or 3 (between benchmark levels 
zero and three). 

K - represents the percentage decrease of ratios between consecutive levels 
going 

from X to Y. 
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AU Subjective Ratio Technique 

This technique involves the use of subjective ratios to assign values to the 
levels between the benchmark levels. This type of ratio assignment requires 
personnel knowledgeable with the criteria, objectives, and workers. Concurrence by 
those being evaluated is desired to ensure the maximum cooperation and 
understanding of the workforce. 



APPENDIX 2 



PROJECT REQUEST LETTER 
INTERVIEW GUIDE 
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Project Request Letter 



* 

* 

* 

* 

Dear 



Enclosed is an explanation of how the Construction Industry Institute is 
continuing its research on design effectiveness. Phase I of the research developed an 
evaluation method and established several design performance criteria. Industrial 
projects were investigated during phase II. In phase III, the researchers wish to 
explore the applications of the method to commercial building and heavy civil 
construction. Included with the general explanation of the evaluation method is a 
copy of the interview guide the researchers will use to gather information from 
individuals on selected projects. 

Please do not let the size of this package trouble you. We are only asking that 
you fill out a point of contact form at this time. 

Your company is being asked to volunteer commercial building and/or heavy 
civil projects which would provide useful information on the design evaluation criteria 
and sub-criteria listed below. Optimally, the projects volunteered would allow the 
researchers to obtain input from the project's owner, designer, and constructor. If 
your company has already implemented the objectives matrix on a project(s) a point of 
contact is requested so the researchers can collect information concerning the strengths 
and weaknesses of the method. For this portion of the research all types of projects 
are welcome. 

Criteria and Sub-Criteria include: 

ACCURACY OF THE DESIGN DOCUMENTS 

— Drawing revisions / total number of drawings 

— Revised drawings / total number of drawings 

— Specification revisions / total number of specifications 
-- Revised specifications / total number of specifications 

— Manhours of rework / total number of manhours 

USABILITY OF THE DESIGN DOCUMENTS 

— Drawing size 

— Number of drawings 

— Number of cross references 

— Field engineering manhours / total engineering manhours 

— Clarity of drawings 

-- Completeness of drawings 
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COST OF DESIGN EFFORT 

— Design cost / construction cost 

— Design manhours expended / design manhours budgeted 

CONSTRUCTABILITY 

— Unrealistic tolerances vs. quality expectations 

— Number of crafts 

— Compatibility with current materials and technology 

ECONOMY OF DESIGN 

— Building layout efficiency 

— Amount of overdesigned members 

— Amount of overspecified materials 

PERFORMANCE AGAINST SCHEDULE 

— Percent of construction document release dates attained 

— Percent of intermediate document release dates attained 

EASE OF OCCUPANCY / START-UP 

— Occupancy preparation days required / days budgeted 

— Number of maintenance personnel required / number budgeted 

The graduate student conducting the interviews is Michael Peek. If you could 
pass this package to an appropriate individual on a suitable project, and return the 
"point of contact" form, Mike will make the next contact. 

RESEARCH OFFICE ADDRESS 
(512) 471-4648 
Michael Peek 

Construction Industry Institute 
The University of Texas at Austin 
3208 Red River Street, Suite 300 
Austin, Texas 78705-2650 

Home Phone Number 
(512) 345-2640 

Again we want to emphasize that this package is purely informative, and 
intended only to give you some insight into what information the research requires. 
Thank you for your time and assistance. 



Sincerely, 



DESIGN RESEARCH PROJECT III - OUTPUTS 
POINT OF CONTACT FORM 
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Please provide the following information regarding the project. All data provided, 
including company identification, will be kept confidential. 



Company name: 

Point of contact: 

Phone number: 

Project title & description: 



Contact's address: 



Please return the completed form to: 



Michael Peek 

Construction Industry Institute 
The University of Texas at Austin 
3208 Red River Street, Suite 300 
Austin, Texas 78705-2650 
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EVALUATION OF DESIGN EFFECTIVENESS 
(DESIGN RESEARCH PROJECT - III) 



Purpose of the Study 

Problem description 

Phases I and II of the "Outputs" research effort have a ) adapted the objectives matrix 
technique as a method for evaluating design effectiveness and b) established 
guidelines for utilizing the method. Unless this method is implemented on actual 
construction projects, no benefits will be realized. 

Problem scope 

The purpose of this research effort is two-fold: (1) to encourage and assist CII 
member companies in implementing the objectives matrix for evaluating design 
effectiveness, and (2) to apply the method to additional areas of construction, such as 
commercial building and heavy civil projects. 

Expected product 

The expected product of this research effort will include reports to the CEI 
membership on the results of implementing the objectives matrix on actual projects. 
Included will be an evaluation of the method's strengths and weaknesses, and 
recommended changes for implementing and using the objectives matrix. 
Additionally, a report will be prepared on the application of the objectives matrix to 
other areas of construction, (only piping was studied in phases I & II of the "outputs" 
research). 
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THE OBJECTIVES MATRIX 



Explanation of the Objectives Matrix 

The following is a brief explanation of the Objectives Matrix. 

Select the performance criteria 

The first step in constructing the matrix is to select the criteria which pertain to the 
performance goals. This requires the identification of key indicators of performance. 
These indicators should relate to such factors as quality, production time, ability to 
meet deadlines, waste, etc. Each of these criterion and how it will be measured 
should be well defined. 

Select performance scores 

Each column in the matrix represents one performance criterion. Each row represents 
a specific level of performance. Performance levels range from a low score of zero, 
to a high score of ten. A score of three is assumed to be the average in order to 
provide more room for improvement. Each score corresponds to a value under the 
criteria, and it is these values which form the performance scale for the criteria. 
Benchmark scores are designated for performance levels zero, three, and ten. 

Level 0 - The lowest level of acceptable performance over a recent period, 
say of two years, under normal operating conditions. 

Level 3 - The current level of performance. 

Level 10 - The desired level of performance. This should represent a level of 
performance that can be realistically obtained in the forseeable 
future with essentially the same resources available at the present 
time. 

Once these benchmark scores are established, the increments between them are 
assigned values. 

Assignment of weights 

Weighting of the performance criteria provides a means of directing management's 
attention toward those criteria which have the most impact on performance. All the 
criteria selected do not have an equal impact on overall performance. A total of one 
hundred points should be distributed among the performance criteria to reflect the 
contribution of each criterion to the overall performance. One aspect of this study is 
to consult with industry personnel to determine what weights should be assigned to 
the chosen criteria. 

Calculate the performance index 

The final phase ties together the criteria scores and weights to determine the overall 
performance index. The performance index is calculated by multiplying the score 
(from zero to ten) corresponding to the measured performance level for each criterion 



by its weight, and adding the products. The figure obtained is entered in the box at 
the bottom right of the matrix, and represents the performance index for a particular 
evaluation period. 
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Objectives Matrix example 

An example of an objectives matrix is attached as an exhibit. 
Evaluation criteria and sub-criteria 



In addition to adapting the objectives matrix for use in the construction industry, 
phase I established seven performance criteria applicable to all types of construction 
projects. Sub-criteria were then developed for each criterion. 

Accuracy of the Design Documents (Accuracy) 

— Drawing revisions / total number of drawings (Drawing Revisions) 

— Revised drawings / total number of drawings (Revised Drawings) 

— Specification revisions / total number of specifications (Specification Revisions) 

— Revised specifications / total number of specifications (Revised Specifications) 

— Manhours of rework / total number of manhours (Rework) 

Usability of the Design Documents (Usability) 

— Drawing size (Drawing Size) 

— Number of drawings ( Number of Drawings) 

— Number of cross references between drawings and specifications (Cross 
References) 

-- Field engineering manhours / total engineering manhours (Field Engineering) 

— Clarity of drawings (Clarity) 

-- Completeness of drawings (Completeness) 

Cost of the Design Effort (Cost) 

— Design cost / construction cost (Design Cost) 

-- Design manhours expended / design manhours budgeted (Design Manhours) 
Constructability (Constructability) 

— Unrealistic tolerances vs. quality expectations (Tolerances) 

— Number of crafts (Different Crafts) 

— Compatibility with current materials and technology (Materials and Technology) 

Economy of Design (Economy) 

— Building layout efficiency (Layout) 

-- Amount of overdesigned members (Oversized Members) 

— Amount of overspecified materials (Overspecified Materials) 

Performance Against Schedule (Schedule) 

-- Percent of construction document release dates attained (Document Release) 

— Percent of intermediate release dates attained (Intermediate release) 
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Ease of Occupancy / Start-up (Occupancy) 

— Occupancy preparation days required / days budgeted (Occupancy Days) 

— Number of maintenance personnel required / number budgeted (Maintenance 
Personnel) 



Sub-Criteria Definitions 



Accuracy: Relates to the two most common design documents, drawings and 
specifications. 

Drawings and Specifications: Refers to all drawings and specifications that are 
necessary to construct the project. 

Revision: Is defined as a change in a drawing or specification that occurs after 
construction of the project has begun. Revisions that occur during the design phase 
or bid phase will not be counted. 

Manhours: Refers to all manhours required for the construction phase of the project. 

Rework: Is defined as a change made to a finished piece, or a portion a finished piece 
of construction. It can also be any additional work performed, but not stipulated on 
the design documents. Rework may be repeated on the same item on more than one 
occasion. Rework occurring only because of design document changes, errors or 
omissions will be counted here. Rework due to an error in a drawing, a change to a 
drawing or a late drawing will be included here. Rework due to a misinterpretation 
of, or conflict with, the design documents will be counted. Rework due to worker 
error will not be counted. 

Usability: Is best reflected by the content and format of the design documents and the 
engineering work required at the construction site. 

Drawing Size: This element is rated on a scale of zero to ten, and should be given a 
high rating when the physical size, scale, and clarity of the drawings were right for 
the work being performed. A low score should be given if the drawing size and scale 
makes them difficult to read. 

Cross References: This item is rated on a scale of zero to ten. If the design 
documents contain an appropriate amount of cross references, that is if all necessary 
referencing between drawings, specifications, and drawings and specifications are 
included, then a high score should be awarded. 

Clarity: Is rated on a scale of zero to ten. This item deals with how well the drawing 
explains what is to be done. If what is to be done is unclear, then a low score should 
be awarded. A drawing that leaves no room for misinterpretation should be given a 
ten. A drawing that could easily mean several different things should be given a zero. 
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Completeness: This item measures on a scale of zero to ten, how well the drawings 
address what is to be done. If the drawing omits several important details, it should 
be given a zero. A drawing that contains all the information necessary to perform the 
work should be given a rating of ten. 

Cost: Cost refers only to those costs necessary to perform the construction phase of 
the project. The total cost of the construction phase includes the cost of the design, 
equipment rental, installation, materials, and labor. 

Engineering Manhours: Manhours spent on actual design. 

Constructability: Is defined as the optimum use of construction knowledge and 
experience in the planning, engineering, procurement, and field operations to achieve 
the overall project objectives. 

Unrealistic Tolerances: This item is rated on a scale of zero to ten, and relates to any 
tolerance that is either unachievable or much too strict for the purpose. Tolerances are 
given a low rating if they are not strict enough to meet the level of quality expected 
for the type of work. 

Crafts: Is rated on a scale of zero to ten, and refers to the number of different crafts 
required for the construction of the project. 

Current Materials and Technology: Is rated on a scale of zero to ten, and includes the 
use of items or methods that are out of date or inefficient. 

Building Layout Efficiency: This item is rated on a scale of zero to ten, and refers to 
how well the building layout is in relation to its function. It takes into account such 
items as placement of stairways, elevators, mechanical rooms etc. 

Overspecified Materials: Is rated on a scale of zero to ten, and includes those 
materials whose properties are more than adequate for their intended use. 

Overdesigned Members: Is rated on a scale of zero to ten, and refers to the number of 
structural members that are much larger than their intended load requires. 

Design Document Release Deadlines: These dates are stipulated in the original 
schedule and include dates on which information, drawings, and specifications are to 
be made available to the constructor. 

Intermediate Release Dates: These dates area deadlines for which drawings and 
specifications must be available for procurement, permits, or other owner or designer 
provided items that pertain to the project. 
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INTERVIEW GUIDE 
FOR 

DESIGN EFFECTIVENESS 
DESIGN RESEARCH PROJECT III - OUTPUTS 



Richard L. Tucker 
Michael A. Peek 



Construction Industry Institute 
The University of Texas at Austin 
3208 Red River Street, Suite 300 
Austin, Texas 78705-2650 
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PROJECT DESCRIPTION SHEET 



1. What type of project is this? 

2. How long were the durations of the design and construction phases of this 
project? 

DESIGN CONSTRUCTION 

3. What type of contracts were used for the design and construction phases of this 
project? 

DESIGN CONSTRUCTION 

4. In what locations were the construction and design phases of this project 
performed? 

DESIGN CONSTRUCTION 

5. What company did the design phase of this project? 

DESIGNER 

6. What company did the construction phase of this project? 

CONSTRUCTOR 

7. What company was the owner of the finished product? How would you rate the 
communications between the owner, the designer, and the construction 
companies? 

OWNER 

Communications OK Problem 

8. Were union or non-union workers involved in the construction of the project? 

9. Was there any overlap between the design and construction of the project? If so, 
how long? 

Overlap Yes No 

Duration (months) 

10. On a scale of one to ten, how would you arbitrarily rate the drawings and 
specifications used on the project? 

1 1 . What was the total cost of this project? 

$ 

12. What other criteria or sub-criteria would you add to the list of criteria and sub- 
criteria discussed in the explanation of the objectives matrix, (as a measurement of 
the design outputs)? What weight would you assign these criteria? 



13. What existing information does your company track that could be incorporated 
into a matrix of this type? 
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ACCURACY OF THE DESIGN DOCUMENTS 



1 . Total number of drawings on this project 
drawings 

2. Total number of revisions to the drawings on this project. 
drawing revisions 

3. Total number of revised drawings on this project. 

revised drawings 

4. Total number of specifications on this project. 

specifications 

5. Total number of revisions to the specifications on this project 
specification revisions 

6. Total number of revised specifications on this project. 

revised specifications 

7. Total number of construction manhours on this project. 
manhours 

8. Total number of construction manhours due to rework on this project. 
manhours 
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USABILITY OF DESIGN DOCUMENTS 



1 . Rating for drawing size for this project (use a scale of zero to ten, three is 
average). 

2. Rating of the number of drawings on this project necessary for crew use (use a 
scale of zero to ten, three is average). 

3. Rating of number of cross references between design documents on this project 
(use a scale of zero to ten, three is average). 

4. Number of field engineering manhours required for this project. 

manhours 

5. Total number of design and field engineering manhours required for this project. 
manhours 

6. Rating of clarity of design documents for this project (use a scale of zero to ten, 
three is average). 

7. Rating of completeness of drawings for this project (use a scale of zero to ten, 
three is average). 
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COST OF DESIGN EFFORT 



1 . Total budgeted manhours for the design of this project. 
manhours 

2. Total expended manhours for the design of this project. 
manhours 

3. Total design cost for this project. 

$ 

4. Total construction cost for this project. 

$ 
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CONSTRUCTABILITY 



1 . Rating for number of different crafts required to complete this project (use a scale 
of zero to ten, three is average). 

2. Rating for the number of unrealistic tolerances versus the quality expectations in 
the design documents for this project (use a scale of zero to ten, three is average). 

3. Rating for level of compatibility of the design of this project with current materials 
and technology (use a scale of zero to ten, three is average). 
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ECONOMY OF DESIGN 



1 . Rating for the amount of overdesigned members in this project (use a scale of zero 
to ten, three is average). 

2. Rating for the amount of overspecified materials in this project (use a scale of zero 
to ten, three is average). 

3. Rating for the level of efficiency in building layout (or project layout for heavy 
civil projects) for this project (use a scale of zero to ten, three is average). 
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PERFORMANCE AGAINST SCHEDULE 



1 . Total number of scheduled release dates for the design documents of this project. 
scheduled release dates 



2. Total number of the design documents released on the scheduled release dates. 
release dates met 



3 . Total number of intermediate scheduled release dates for the design documents of 
this project. 

intermediate release dates 



4 . 



Total number of intermediate scheduled release dates met as per the scheduled 
dates. 

release dates met 
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EASE OF OCCUPANCY / START-UP 



1 . Total number of days required to prepare for occupancy / use after construction 
was completed. 

days 

2. Total number of days budgeted to prepare for occupancy / use. 

days 

3. Total number of maintenance personnel manhours required to prepare for 
occupancy / use of the building / structure. 

manhours 

4. Total number of maintenance personnel manhours budgeted to prepare for 
occupancy / use of the building / structure. 

manhours 
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WEIGHT ESTIMATES FOR CRITERIA AND SUB 

EVALUATION OF DESIGN CRITERIA 
RANK 

Accuracy of the design documents 

Usability of the design documents 

Cost of design effort 

Constructability 

Economy of design 

Performance against schedule 

Ease of occupancy / start-up 



EVALUATION OF DESIGN SUB-CRITERIA 
RANK 

ACCURACY OF THE DESIGN DOCUMENTS 

Drawing revisions / total number of drawings 

Revised drawings / total number of drawings 

Specification revisions / total number of specifications 

Specifications revised / total number of specifications 

Manhours of rework / total number of manhours 



USABILITY OF THE DESIGN DOCUMENTS 



CRITERIA 



WEIGHT 



SUM = 100 



WEIGHT 



SUM = 100 



Drawing size 
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RANK 



WEIGHT 

USABILITY (continued) 

Number of drawings 

Number of cross references 

Field engineering manhours / total engineering manhours 

Clarity 

Completeness of drawings 

SUM = 100 



COST OF DESIGN EFFORT 

Design cost / construction cost 

Design manhours expended / design manhours budgeted 

SUM = 100 



CONSTRUCTABILITY 

Unrealistic tolerances vs. quality expectations 

Number of crafts 

Compatibility with current materials and technology 

SUM = 100 



ECONOMY OF DESIGN 

Building layout efficiency 

Amount of overdesigned members 

Amount of overspecified materials 

SUM = 100 
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RANK 



WEIGHT 



PERFORMANCE AGAINST SCHEDULE 
Percent of construction document release dates attained 
Percent of intermediate release dates attained 



SUM = 100 



EASE OF OCCUPANCY / START-UP 

Occupancy preparation days required / days budgeted 

Number of maintenance personnel required / number budgeted 

SUM = 100 
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ESTIMATES OF INDUSTRY AVERAGES 
FOR EVALUATION CRITERIA 

10 - Optimal 3 - Current performance 0 - Minimum acceptable 

ACCURACY OF THE DESIGN DOCUMENTS 
Drawing revisions / total number of drawings 
10. - 3.~ 0.- 

Revised drawings / total number of drawings 
10.- 3.- 0.- 

Specification revisions / total number of specifications 
10.- 3.- 0.- 

Revised specifications / total number of specifications 
10.- 3.- 0.- 

Manhours for rework / total number of construction manhours 
10.- 3.- 0.- 

US ABILITY OF THE DESIGN DOCUMENTS 
Field engineering manhours / total engineering manhours 



10 .- 



3.- 



0 .- 



COST OF DESIGN EFFORT 



Design cost / construction cost 



10 .- 



3.- 



0 .- 



Design manhours expended / design manhours budgeted 



10.- 



3.- 



0 .- 
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PERFORMANCE AGAINST SCHEDULE 
Percent of construction document release dates attained 
10 .- 3 .~ 0 .~ 

Percent of intermediate release dates attained 
10 .- 3 .- 0 .~ 

EASE OF OCCUPANCY / START-UP 

Occupancy / start-up preparation time required / preparation time budgeted 
10 .- 3 .- 0 .- 

Number of maintenance personnel required / number budgeted 
10 .- 3 .- 0 .- 
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